Enrico Marchesini

I am a Postdoctoral Associate at Massachusetts Institute of Technology advised by Prof. Priya Donti, in the Laboratory for Information & Decision Systems. Previously I was a Postdoctoral Associate with Prof. Christopher Amato at Northeastern University, and a Ph.D. student at the University of Verona, with Prof. Alessandro Farinelli.

My research interests are driven by the impact that Deep Reinforcement Learning (RL) could have in real-world domains, where effective exploration, safety, and asynchronous execution are key requirements for autonomous learning agents. Hence, I focus on developing Deep RL algorithms aimed at tackling these fundamental challenges in simulation, multi-agent systems, and realistic applications. I am currently working on safe and asynchronous multi-agent RL and will focus on its applications to power systems during my Postdoc.

Outside of work, I spend time climbing and hiking.

News

2024	August Our preprint “On Stateful Value Factorization in Multi-Agent Reinforcement Learning” addresses an important problem that has been overlooked in the MARL community for more than half a decade! June Our preprint “Improving Policy Optimization via ε-Retrain” investigates how to improve policy optimization methods using restart distributions and is out! January Our paper “Enumerating Safe Regions in Deep Neural Networks with Provable Probabilistic Guarantees” has been accepted at AAAI 2024. Congratulations to Luca Marzari for leading the project! What a great start to 2024! Another collaboration with the colleagues from Oregon State University, Alp Aydeniz, and Kagan Tumer has been accepted at AAMAS 2024. Check out our abstract “Entropy Seeking Constrained Multiagent Reinforcement Learning” during the conference in May!
2023	September Excited to share our first collaboration with Alp Aydeniz, Robert Loftin, and Kagan Tumer has been accepted at MRS 2023. Check out “Entropy Maximization in High Dimensional Multiagent State Spaces” during the conference in December! May I’m thrilled to announce my new adventure as a Postdoctoral researcher at MIT; more details will come in November! Working with Chris Amato has been great and I’m looking forward to continuing our collaborations! January Our paper “Improving Deep Policy Gradients with Value Function Search” has been accepted at ICLR 2023. Our paper “Safe Deep Reinforcement Learning by Verifying Task-Level Properties” has been accepted at AAMAS 2023. Our paper “Online Safety Property Collection and Refinement for Safe Deep Reinforcement Learning in Mapless Navigation” has been accepted at ICRA 2023.
2022	July Presenting “Safety-Informed Mutations for Evolutionary Deep Reinforcement Learning” at Gecco EvoRL Workshop. April Excited to join Prof. Christopher Amato as a Postdoctoral researcher at Northeastern University. January Our paper “Enhancing Deep Reinforcement Learning Approaches for Multi-Robot Navigation via Single-Robot Evolutionary Policy Search” has been accepted at ICRA 2022.
2021	November Our paper “Exploring Safer Behaviors for Deep Reinforcement Learning” has been accepted at AAAI 2022 (15% acceptance rate). June Three papers have been accepted at IROS 2021: “Centralizing State-Values in Dueling Networks for Multi-Robot Reinforcement Learning Mapless Navigation.” “Benchmarking Safe Deep Reinforcement Learning in Aquatic Navigation.” “Safe Reinforcement Learning using Formal Verification for Tissue Retraction in Autonomous Robotic-Assisted Surgery.”

Selected publications

On Stateful Value Factorization in Multi-Agent Reinforcement Learning

Enrico Marchesini, Andrea Baisero, Rupali Bhati, and 1 more author

In arXiv, 2024

Abs Bib HTML

Value factorization is a popular paradigm for designing scalable multi-agent reinforcement learning algorithms. However, current factorization methods make choices without full justification that may limit their performance. For example, the theory in prior work uses stateless (i.e., history) functions, while the practical implementations use state information – making the motivating theory a mismatch for the implementation. Also, methods have built off of previous approaches, inheriting their architectures without exploring other, potentially better ones. To address these concerns, we formally analyze the theory of using the state instead of the history in current methods – reconnecting theory and practice. We then introduce DuelMIX, a factorization algorithm that learns distinct per-agent utility estimators to improve performance and achieve full expressiveness. Experiments on StarCraft II micromanagement and Box Pushing tasks demonstrate the benefits of our intuitions.
title = {On Stateful Value Factorization in Multi-Agent Reinforcement Learning}, author = {Marchesini, Enrico and Baisero, Andrea and Bhati, Rupali and Amato, Christopher}, booktitle = {arXiv}, year = {2024}, }
Improving Policy Optimization via ε-Retrain

Luca Marzari, Changliu Liu, Priya Donti, and 1 more author

In arXiv, 2024

Abs Bib HTML

We present ε-retrain, an exploration strategy designed to encourage a behavioral preference while optimizing policies with monotonic improvement guarantees. To this end, we introduce an iterative procedure for collecting retrain areas – parts of the state space where an agent did not follow the behavioral preference. Our method then switches between the typical uniform restart state distribution and the retrain areas using a decaying factor ε, allowing agents to retrain on situations where they violated the preference. Experiments over hundreds of seeds across locomotion, navigation, and power network tasks show that our method yields agents that exhibit significant performance and sample efficiency improvements. Moreover, we employ formal verification of neural networks to provably quantify the degree to which agents adhere to behavioral preferences.
title = {Improving Policy Optimization via \epsilon-Retrain}, author = {Marzari, Luca and Liu, Changliu and Donti, Priya and Marchesini, Enrico}, booktitle = {arXiv}, year = {2024}, }
Improving Deep Policy Gradients via Value Function Search

Enrico Marchesini, and Christopher Amato

In International Conference on Learning Representations (ICLR), 2023

Abs Bib HTML

Deep Policy Gradient (PG) algorithms employ value networks to drive the learning of parameterized policies and reduce the variance of the gradient estimates. However, value function approximation gets stuck in local optima and struggles to fit the actual return, limiting the variance reduction efficacy and leading policies to sub-optimal performance. In this paper, we focus on improving value approximation and analyzing the effects on Deep PG primitives such as value prediction, variance reduction, and correlation of gradient estimates with the true gradient. To this end, we introduce a Value Function Search that employs a population of perturbed value networks to search for a better approximation. Our framework does not require additional environment interactions, gradient computations, or ensembles, providing a computationally inexpensive approach to enhance the supervised learning task on which value networks train. Crucially, we show that improving Deep PG primitives results in improved sample efficiency and policies with higher returns using common continuous control benchmark domains.
@inproceedings{ICLR2023_vfs, title = {Improving Deep Policy Gradients via Value Function Search}, author = {Marchesini, Enrico and Amato, Christopher}, booktitle = {International Conference on Learning Representations (ICLR)}, year = {2023}, url = {https://openreview.net/forum?id=6qZC7pfenQm}, }